Efficient Image Captioning for Edge Devices

Recent years have witnessed the rapid progress of image captioning. However, demands for large memory storage and heavy computational burden prevent these captioning models from being deployed on mobile devices. The main obstacles lie in heavyweight visual feature extractors (i.e., object detectors) complicated cross-modal fusion networks. To this end, we propose LightCap, a lightweight captioner resource-limited core design is built recent CLIP model efficient be specific, one hand, leverage to extract compact grid features without relying time-consuming detectors. On other transfer image-text retrieval scenarios by devising novel concept extractor modulator. We further optimize parallel prediction heads via sequential ensemble distillations. With carefully designed architecture, our merely contains 40M parameters, saving size more than 75% FLOPs 98% comparison with current state-of-the-art methods. In spite low capacity, still exhibits performance prevalent datasets, e.g., 136.6 CIDEr COCO Karpathy test split. Testing smartphone only single CPU, proposed LightCap fast inference speed 188ms per image, which ready practical applications.